Modular architecture for computer network operations

ABSTRACT

Disclosed embodiments include a method of operation of a distributed network system. The method includes nodes of the network system that send messages over a protocol-independent message bus, and other nodes that receive the messages. Content from the received messages can be stored in a database distributed among nodes of the network system. At least some of the content stored in the database is published. The published content can be accessed by one or more applications to perform one or more functions.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of commonly assigned U.S.Provisional Patent Application Ser. No. 62/333,602, filed on May 9,2016, and entitled “MODULAR ARCHITECTURE FOR COMPUTER NETWORK,” thedisclosure of which is hereby incorporated herein in its entirety byreference.

TECHNICAL FIELD

The disclosed teachings relate generally to computer networks. Moreparticularly, the disclosed teachings relate to a modular distributednetwork application framework.

BACKGROUND

A routing protocol controls how nodes in a computer network systemcommunicate with each other. The nodes are connected by links to form atopology in which the nodes disseminate messages over the network. Theblock diagram of FIG. 1 illustrates a conventional routing protocolarchitecture 10. The routing protocol architecture 10 includes acomponent for message communications 12, a database 14, and a routingalgorithm 16. Messages are disseminated throughout the network of nodesin accordance with a process of the message communications 12.

The disseminated messages are stored in the database 14, distributedamong the network of nodes. The content of the database 14 is used bythe routing algorithm 16 to calculate routing paths, which arecollectively stored in each node's routing table. The format, exchange,and synchronization of messages, as well as routing calculations, areall tightly integrated with the routing protocol architecture 10.

For example, link-state routing protocols, such as open shortest pathfirst (OSPF) and intermediate system to intermediate system (IS-IS),include processes performed by each node to construct a topological map.The messages can be propagated over the network by using flooding, whereevery incoming packet is sent through every outgoing link except the oneit arrived on. In this example, the routing protocols include adistributed database that stores link-state and prefix data used by eachnode to calculate paths to every destination in the network. Thecollection of paths forms the node's routing table.

Thus, the nodes use proprietary routing protocols to run rigid processesfor message communications, synchronization, and calculating routingpaths. The protocols are limited to performing routing functions andcannot scale well in networks that exceed 1,000 nodes because floodingscales poorly due to significant overhead, especially in meshed wirelessnetworks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a routing protocol architecture;

FIG. 2 illustrates a distributed network system topology according tosome embodiments of the present disclosure;

FIG. 3 is a block diagram that illustrates a modular architectureimplemented in a network according to some embodiments of the presentdisclosure;

FIG. 4 is a flowchart illustrating a process for disseminating messagesin a network according to some embodiments of the present disclosure;

FIG. 5 is a flowchart illustrating a process for updating andsynchronizing a database of the architecture according to someembodiments of the present disclosure;

FIG. 6 is a flowchart illustrating a process for mitigating theundesirable effects of flooding according to some embodiments of thepresent disclosure;

FIG. 7 illustrates a spanning tree topology according to someembodiments of the present disclosure;

FIG. 8 is a sequence diagram illustrating a process performed by a helloprotocol module according to some embodiments of the present disclosure;

FIG. 9 is a flowchart illustrating a process performed by a securitymodule of the architecture according to some embodiments of the presentdisclosure; and

FIG. 10 is a block diagram of a computer operable to implement thedisclosed technology according to some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The embodiments set forth below represent the necessary information toenable those skilled in the art to practice the embodiments, andillustrate the best mode of practicing the embodiments. Upon reading thefollowing description in light of the accompanying figures, thoseskilled in the art will understand the concepts of the disclosure andwill recognize applications of these concepts that are not particularlyaddressed herein. It should be understood that these concepts andapplications fall within the scope of the disclosure and theaccompanying claims.

The purpose of terminology used here is only for describing embodimentsand is not intended to limit the scope of the disclosure. Where contextpermits, words using the singular or plural form may also include theplural or singular form, respectively.

As used herein, unless specifically stated otherwise, terms such as“processing,” “computing,” “calculating,” “determining,” “displaying,”“generating” or the like, refer to actions and processes of a computeror similar electronic computing device that manipulates and transformsdata represented as physical (electronic) quantities within thecomputer's memory or registers into other data similarly represented asphysical quantities within the computer's memory, registers, or othersuch storage medium, transmission, or display devices.

As used herein, the terms “connected,” “coupled,” or variants thereof,refer to any connection or coupling, either direct or indirect, betweentwo or more elements. The coupling or connection between the elementscan be physical, logical, or a combination thereof.

The disclosed embodiments include an architecture that effectivelytransforms a routing protocol into a modular application developmentframework. The components of the framework are decoupled to enableindependent operations. The disclosed embodiments include processesperformed by a network that is segmented to enable efficientdissemination of messages. In some embodiments, code runs on each nodeof a network to disseminate messages (e.g., exchange and/or propagatemessages) across the network of nodes. The term “message,” as referredto herein, includes data such as a network packet and can be usedinterchangeably with the terms “data” or “information” unless thecontext suggests otherwise.

The architecture is agnostic to network topology, underlying transportprotocols, and message payloads. Hence, a message is not limited to anyspecific format. The architecture includes modules that separatedistribution, synchronization, and use of messages. These modules areindependent functional components that can be optimized separately. Assuch, the architecture is flexible, scalable, and extensible to provideadditional functionality over existing architectures that are limitedonly to routing protocols.

In particular, introduced here are at least one method, at least onesystem, and a node of the system. The at least one method includes amethod of operation of a distributed network system. The method includesnodes of the network sending messages over a protocol-independentmessage bus, and other nodes receiving the sent messages. Content fromthe received messages can be stored in a database distributed amongnodes of the network system. At least some of the content stored in thedatabase is published. The published content can be accessed byapplications to perform one or more functions.

In some embodiments, a distributed network application frameworkarchitecture includes a protocol-agnostic message bus connected to nodesof the network. The nodes can be configured to facilitate sending,receiving, or propagating messages over the protocol-agnostic messagebus. The architecture can include a payload-agnostic databasedistributed across the nodes and storing content communicated over themessage bus, and includes one or more applications utilizing the contentstored in the database to perform one or more functions.

In some embodiments, a node of a distributed network system can includean interface to a protocol-agnostic message bus connected to nodes ofthe network system.

The interface can facilitate sending, receiving, or propagating messagesover the protocol-agnostic message bus. The node includes a databasecomponent of a payload-agnostic database distributed across the nodes ofthe network system. The database component can store content receivedover the protocol-agnostic message bus. The node can include one or moreapplications that use the content stored in the database component toperform one or more functions.

The disclosed architecture can implement, among others, functions of alink-state routing protocol, where such function can be optimizedseparately to allow for scaling in larger networks compared toconventional link-state networks. In contrast, existing routingprotocols do not include component parts that can be optimizedindependently. Instead, the routing protocols themselves would need tobe changed to perform optimization.

FIG. 2 illustrates a distributed network system topology according tosome embodiments of the present disclosure. The network 18 includesnodes 20 (collectively referred to as nodes 20 and individually as nodes20-1 through 20-6). The nodes 20 may include any type of computingdevices (e.g., server) that can provide a variety of services and/ordata. The nodes 20 are interconnected by links 22. The links 22 mayinclude any combination of private, public, wired, or wireless portions.Data communicated over the links 22 may be encrypted or unencrypted atvarious locations and along portions of the network 18.

The network 18 could be implemented as a backhaul, datacenter, backbone,and the like. For example, the network 18 could be implemented in awireless backhaul that integrates the nodes 20. Further, the network 18is not limited to the number of nodes 20 shown in FIG. 2. For example,the network 18 can include more than 1,000 or 2,000 nodes, which isgreater than the maximum number of nodes in typical networks thatimplement conventional routing protocols.

Each component of the network 18 may include combinations of hardwareand/or software to process data, perform functions, communicate over thenetwork 18, and the like. For example, any component may include aprocessor, memory or storage, a network transceiver, a display,operating system and application software, and the like. Othercomponents, hardware, and/or software included in the network 18 arewell known to persons skilled in the art and, as such, are not shown ordiscussed herein.

FIG. 3 is a block diagram that illustrates a modular architecture 24implemented in the network 18 according to some embodiments of thepresent disclosure. The architecture 24 includes a message bus 26 and akey-value database 28 (hereinafter “database 28”) as standalonecomponents. In other words, the database 28 can operate independently ofthe message bus 26. The database 28 can be distributed among any of thenodes 20. As such, for example, each node 20 can have a database 28component. Functionality can be built on top of these standalonecomponents as applications 30 that use data published by the nodes 20,which is stored as the content in the database 28. For example, thepublished data can be used to perform functions, such as an addressassignment function, an auditing function, a monitoring function, anadaptive control function, an optimization function, a machine learningfunction, or combinations thereof.

To aid in understanding, the architecture 24 can be viewed as adisaggregation and generalization of a routing protocol architectureinto separate components, which allows for use and re-use of datapublished by the nodes 20, and stored in the database 28, forapplications such as a routing application 30-1, a bandwidth allocationapplication 30-2, a traffic engineering application 30-3, and any otherapplications 30-4. In some embodiments, the applications 30 can beresident on any of the nodes 20 of the network 18 or nodes separate anddistinct from the network 18.

In some embodiments, the message bus 26 can be implemented in a meshnetwork where each node cooperates to distribute messages in thenetwork. In some embodiments, control of the architecture 24 can becentralized or distributed across the network 18. For example, acentralized controller could operate to push the messages over thenetwork 18, and nodes may engage in distributed communications tocontrol the network 18.

FIG. 4 is a flowchart illustrating a process 400 for disseminatingmessages in the network 18 according to some embodiments of the presentdisclosure. In step 402, messages are sent by nodes of the network 18over the message bus 26. For example, a subset of the nodes 20 can sendmessages with nodes 20 over the message bus 26. The message bus 26 isprotocol-agnostic and payload-agnostic to carry the messages publishedby nodes 20 in any format. The message bus 26 includes a common datamodel, a common command set, and a messaging infrastructure to allowcommunications through shared interfaces. As such, any data can be sentanywhere in the network 18 over the message bus 26 without needing to beinterpreted by any or all intermediate nodes.

The message bus 26 can be built by connecting the nodes 20 usingtransmission control protocol (TCP) connections in a peer-to-peer (P2P)fashion. The message bus 26 uses a software framework that providesscalable cross-language services. Specifically, the message bus 26 usesa common encoding for all data that is communicated over the message bus26. An example of the software framework is Thrift, which is aninterface definition language and binary communication protocol that canbe used to define and create services for numerous languages.

The software framework can be used as a remote procedure call (RPC)framework and combines a software stack with a code generation engine tobuild services that work efficiently and seamlessly between C#, C++,Cappuccino, Cocoa, Delphi, Erlang, Go, Haskell, Java, Node.js, OCaml,Perl, PHP, Python, Ruby, Smalltalk, and other languages. Using thesoftware framework to encode all data simplifies coding over routingprotocols that define their own encoding. Other software frameworks thatcan be used include JSON, which is a language-independent data format,XML, and the like.

In step 404, nodes of the network 18 can propagate messages over themessage bus 26 to other nodes of the network 18. For example, at leastsome of the messages sent from a subset of the nodes 20 can bepropagated by another subset of the nodes 20 over the network 18.Flooding can be used as a common approach to propagate messages over thetopology of nodes 20. Detailed further below are techniques that curtailthe undesirable effects of flooding (e.g., excessive overhead) toimprove scalability.

In step 406, nodes of the network 18 receive the messages sent and/orpropagated by other nodes of the network 18. For example, a subset ofthe nodes 20 can receive messages that were sent and/or propagated byother subsets of the nodes 20. In this context, a node that is said to“receive” a message is a node that consumes the content of the message,whereas a node that is said to “propagate” a message is a node thatrelays the message without necessarily consuming its content. As such,any of the nodes 20 can send, receive, and/or propagate messages.

In step 408, content included in the disseminated messages is stored inthe database 28 (e.g., consumed). The database 28 istopology-independent and decoupled from an underlying protocol. The “keyvalues” of the database 28 refer to a data storage paradigm designed forstoring, managing, and publishing objects that have many differentfields within them, each containing data. These objects are stored andretrieved using a key that uniquely identifies the object and is used tofind the data within the key-value database. The database 28 of thedisclosed architecture may have no limit on key or value size.

In step 410, at least some of the content stored in the database 28 ispublished such that it is made available to the applications 30. Thedatabase 28 stores any type of content as keys and values. The contentis encoded as objects using a scalable cross-language software framework(e.g., Thrift encoded objects). Lastly, in step 412, the applications 30can independently use and re-use the content stored in the database 28to perform a variety of functions, in addition to or instead of routing.Examples of the variety of functions include an address assignmentfunction, an auditing function, a monitoring function, an adaptivecontrol function, an optimization function, a machine learning function,or any other computation that could be performed in a distributedmanner.

In some embodiments, content stored in the database 28 can be updatedwith the payload of the disseminated messages. For example, existingcontent of the database 28 can be synchronized with new content includedin the disseminated messages. In some embodiments, the existing contentin the database 28 can be associated with one or more timestamps. Theexisting content can be designated as expired (i.e., stale) content uponexceeding a period of time stored in the database 28 relative to the oneor more timestamps. The stale content can be removed or replaced withnew content from the disseminated messages. In some embodiments,existing content stored in the database 28 may not expire, and newcontent can replace the existing content upon exceeding a thresholdstorage limit of the database 28. Hence, the database 28 can becontinuously updated with new content from disseminated messages as ifthe database has an infinite storage capacity. As such, the database 28can be continuously updated with new content for use and re-use byapplications to perform various functions.

In another embodiment, a process for updating the database 28 may employeviction logic. For example, FIG. 5 is a flowchart illustrating aprocess 500 for updating and synchronizing the database 28 according tosome embodiments of the present disclosure. In step 502, new content isobtained from the payload of messages received over the message bus 26among the nodes 20. In step 504, database logic can determine whethereviction of existing content is required to store the new content (e.g.,keys and values). If eviction is not required, the new content is simplystored in the database 28.

In step 506, if eviction is required, a cache algorithm can be used toidentify content that is least recently used (LRU) by applications ascandidates for the eviction. In step 508, the LRU content is removedfrom the database 28. Lastly, in step 510, the database 28 stores thenew content in entries that contained the now-evicted LRU content.

The database 28 is not purely event driven. Instead, the database 28 mayimplement an anti-entropy protocol where database components of adjacent(directly connected) nodes 20 periodically synchronize their contents.This synchronization allows the architecture 24 to avoid global networkstate refresh operations (state aging and re-flooding) that otherwiseoccur in IS-IS and OSPF routing protocols.

As indicated above, having the message bus 26 and the database 28decoupled from the applications 30 improves the extensibility of thearchitecture 24. In particular, applications other than a mere routingapplication can use the content from messages distributed over thenetwork 18. In contrast, introducing a new application (other thanrouting) in traditional systems may be impossible without adding a newunderlying protocol. Yet the architecture 24 can add any applications tothe network 18 without requiring additional protocols because, forexample, the message bus 26 can exchange data in any format.

Thus, embodiments of the architecture 24 are not limited to a networkrouting application. Instead, the architecture 24 could send any dataover the network 18 and publish that data for use by any of theapplications 30. The applications 30 can operate independently of themessage bus 26, the database 28, and each other. In fact, only theapplications 30 need to interpret the content from the messages. Thatis, the message bus 26 and database 28 do not need to interpret thecontent. Any number and type of applications 30 can be built on top ofthe message bus 26 and the database 28. The applications 30 can use andre-use the stored content to perform associated functions, such as, forexample, layer 3 routing, performance-based routing, multiprotocol labelswitching, and the like.

For example, the routing application 30-1 could include a link-staterouting algorithm that uses link-state data published by every node 20and stored in the database 28. The routing algorithm could use thelink-state data to compute routing paths. As such, the routingapplication 30-1 uses information obtained by nodes 20 and publishedglobally to obtain a network graph. The architecture 24 allows otherapplications to use or re-use the published data for other functions.

For example, the bandwidth allocation application 30-2 could computefair queueing weights for network topologies where traffic flows to orfrom a single network exit or entry point. The computation is based onthe measurements and global publications of the live traffic closer tothe edge nodes 20 of the network 18 where traffic consumers reside.Thus, the architecture 24 can build fair traffic distribution model inthe network 18. Hence, the architecture 24 is extensible to allownumerous and diverse applications 30 to be built on the message bus 26and database 28 to perform functions that are otherwise not readilyavailable in existing architectures.

As indicated above, messages can be propagated by some of the nodes 20over the network 18 by using flooding, which cannot scale well due toexcessive flooding overhead. The architecture 24 overcomes thesedrawbacks by implementing a combination of techniques that curtail theundesirable effects of flooding.

FIG. 6 is a flowchart illustrating a process 600 for mitigating theundesirable effects of flooding when propagating messages according tosome embodiments of the present disclosure. In step 602, varioustechniques are implemented to segment the network 18 into differentgroups of nodes 20. As such, flooding overhead can be mitigated bypreventing messages from repeatedly traversing the same segment.Examples of these techniques include spanning trees, clustering, andpath tracing.

In step 604, spanning trees are computed over segments of the network18. Each segment of the network 18 covered by a single spanning tree canbe considered a cluster. For example, a spanning tree algorithm can bedistributed across the nodes 20. The distributed spanning tree algorithmcan run using a hello protocol to compute the spanning trees, asdetailed further below. The spanning tree algorithm computes logicalloop-free topologies, including one or more spanning trees.

FIG. 7 illustrates a spanning tree topology 32. The spanning treetopology 32 has a single active path 38 between any two nodes 36. Thespanning tree topology 32 can constitute a cluster. In someconfigurations, portions of the spanning tree topology 32 can formdifferent clusters. As such, the spanning tree algorithm can performclustering of the nodes 20 in the network 18. The network 18 can besegmented into a number of distinct clusters ranging from one to thetotal number of nodes 20. For example, the entire network 18 couldconstitute a single cluster, or each node 20 could constitute a singlecluster.

Referring back to FIG. 6, in step 604, messages distributed within acluster can follow the spanning tree loop-free topology, and messageflooding can occur between clusters. Clusters could be used to controlmessage exchange such that a message is prevented from traversing thesame cluster. This is particularly beneficial in network topologies thatare uniform, such as datacenters, to improve the efficiency of dataexchange.

Path tracing can be applied to limit the flooding. Path tracing is aprocess used to determine the path taken by a message propagated in anetwork. In some embodiments, the process of path tracing includes adata format defined by the message bus 26. The format could have customfields that include path information such as a node or a group of nodesthat was traversed by a message. Information about the path taken by amessage can be used to prevent a message from traversing the same nodeor group of nodes in a flooding exchange.

For example, in step 608, a node can drop a packet that has alreadytraversed that node or traversed a group of nodes to which the nodebelongs (e.g., cluster). For example, messages can be appended withcluster identifiers as they traverse clusters in the network 18. A nodecan drop a message that already traversed its cluster. Moreover, everymessage sent by a node 20 can include a “hop count,” which permits thenetwork 18 to drop messages that are caught in transient routing loops.As such, flooding overhead is curtailed by limiting flooding in thenetwork 18.

Scaling is improved by segmenting the network 18 because floodingoverhead is curtailed. For example, networks can be scaled to include8,000 or more nodes. The number of clusters that include spanning treescould be calculated to provide a desired scalability by reducingoverhead in the network. In some embodiments, the network 18 could beconfigured to flood information in a similar manner as OSPF/IS-IS withadditional flooding loop checks via path tracing. Further, segmentingcan also protect against flooding storms, and would improve theefficiency of applications 30 using content in the database 28 obtainedvia flooding.

As such, for example, the combination of spanning-tree forwarding withinclusters and path-tracing between clusters used to drop messages reducesflooding overhead, yet maintains efficient message propagation in thenetwork 18, in contrast to using conventional flooding, which has thegreatest propagation delay but maximum overhead. In contrast, a singleglobal spanning tree would have the least overhead, but the longestmessage propagation delays.

The architecture 24 can include modules used to build and maintain anetwork of nodes. For example, a hello protocol module can performprocesses for establishing and maintaining relationships betweenneighboring nodes. The hello protocol module enables bidirectional dataexchange between neighboring nodes to discover nodes newly connected tothe network 18, detect changes of the nodes in the network 18, andcalculate metrics for optimizing flooding. The hello protocol moduleruns independently of the message bus 26.

FIG. 8 is a sequence diagram 800 that illustrates a hello protocolprocess between neighboring nodes according to some embodiments of thepresent disclosure. The disclosed hello protocol involves software coderunning on each node 20 of the network 18, which initiates a floodingexchange of a “hello message” sent by a node 20. The hello messages canbe sent across the network 18 over connecting links. The hello messagescould be sent in-band with other messages, or out-of-band.

In step 802, the node 20-1 sends a hello message to a neighboring node20-2. In step 804, the node 20-2 receives the hello message from node20-1. In step 806, the node 20-2 responds back to node 20-1 with anacknowledgement that it received the hello message. In step 808, thenode 20-2 sends the received hello message to the neighboring node 20-3.In step 810, the node 20-3 receives the hello message from node 20-2. Instep 812, the node 20-3 responds to node 20-1 with an acknowledgementthat it received the hello message. The node 20-3 subsequently continuesto flood the network 18 of nodes 20-N. The receiving nodes respond backto the originating node 20-1, and sends the hello message forward toother nodes 20.

Each node 20 can run a process of the hello protocol to discoverneighboring nodes and then possibly create transport connections for themessage bus 26. For example, the hello process can be used by a new nodeto announce its presence in a network. For example, the node 20-1 may bea new node, and nodes 20-2 through 20-N may be existing nodes of thenetwork 18. The hello message may include the sending node's address forforming a connection and a discovery requirement. In some embodiments,hello messages can also be sent out periodically by all nodes 20-1through 20-N of the network to confirm existing connections. Flooding ofthe hello messages can also be limited by path tracing, spanning trees,clustering, and the like.

The hello protocol process of FIG. 8 can be used to compute spanningtree metrics and organize the network into clusters, in addition todiscovering neighboring nodes. For example, in step 814, the node 20-1can compute spanning tree metrics based on the hello messaging. Forexample, spanning tree metrics and clustering can be computed using adistributed algorithm running on the nodes 20 based on node-to-nodelatency measurements obtained from acknowledgements received from nodes20-2 through 20-N. As such, the hello protocol could be used for runningthe spanning-tree algorithm to automate clustering.

As shown, the hello protocol process establishes bidirectionalcommunications between neighboring nodes. In particular, the helloprotocol process could be used to exchange data between nodes 20 in thesame link 22, without initiating global flooding. Moreover, the helloprotocol could be used for “lazy flooding” to propagate someconfiguration parameters that control the message bus 26 behavior over asubset of the nodes 20 of the network 18. Moreover, the hello protocolincludes functionality to provide bidirectional neighbor discovery. Forexample, an existing node that broadcasts a hello message can discover anew node that responded to the hello message.

The architecture 24 can include a security module to authenticate nodesand encrypt messages communicated over the message bus 26. In contrast,traditional routing protocols provide for authenticating communicationsonly by using keyed hash-based signatures. For example, in someembodiments, public-key security is used to authenticate hello messages,and private/public-key encryption/decryption is used for all messagescommunicated over the message bus 26 between neighboring nodes 20.

FIG. 9 is a flowchart illustrating a process 900 performed by a securitymodule of the architecture 24 according to some embodiments of thepresent disclosure. The security module of the architecture 24 mayoperate the nodes 20 in different modes, including a learning mode and aprotected mode. In step 902, a determination is made about whether thenodes 20 are in a learning mode. In step 904, while in the learningmode, the nodes 20 learn of public keys of other nodes. Nodes ofcorresponding public keys learned while the nodes 20 are in the learningmode are deemed as trusted nodes. Otherwise, in step 908, only messagesfrom trusted nodes are authenticated by the nodes.

In step 906, a determination is made about whether the nodes 20 arerunning in a protected mode. In step 908, while the nodes 20 are inprotected mode, only messages from trusted nodes are authenticated bythe security module. Otherwise, the process 900 returns to thebeginning. As such, the information required for authentication isdistributed among nodes in a manner that avoids the complicated publickey distribution process of conventional systems.

FIG. 10 is a block diagram of a computer 40 (e.g., node 20) of thenetwork 18 operable to implement the disclosed technology according tosome embodiments of the present disclosure. The computer 40 may be ageneric computer or specifically designed to carry out features of thenetwork 18. For example, the computer 40 may be a System-On-Chip (SOC),a Single-Board Computer (SBC) system, a desktop or laptop computer, akiosk, a mainframe, a mesh of computer systems, a handheld mobiledevice, or combinations thereof.

The computer 40 may be a standalone device or part of a distributedsystem that spans multiple networks, locations, machines, orcombinations thereof. In some embodiments, the computer 40 operates as aserver computer or a client device in a client-server networkenvironment, or as a peer machine in a peer-to-peer system. In someembodiments, the computer 40 may perform one or more steps of thedisclosed embodiments in real time, near real time, offline, by batchprocessing, or combinations thereof.

The computer 40 includes a bus 42 operable to transfer data betweenhardware components. These components include a control 44 (i.e.,processing system), a network interface 46, an Input/Output (I/O) system48, and a clock system 50. The computer 40 may include other componentsnot shown nor further discussed for the sake of brevity. One havingordinary skill in the art will understand any hardware and softwareincluded but not shown in FIG. 10.

The control 44 includes one or more processors 52 (e.g., CentralProcessing Units (CPUs), Application Specific Integrated Circuits(ASICs), and/or Field Programmable Gate Arrays (FPGAs)) and memory 54(which may include software 56). The memory 54 may include, for example,volatile memory such as random-access memory (RAM) and/or non-volatilememory such as read-only memory (ROM). The memory 54 can be local,remote, or distributed.

A software program (e.g., software 56), when referred to as “implementedin a computer-readable storage medium,” includes computer-readableinstructions stored in a memory (e.g., memory 54). A processor (e.g.,processor 52) is “configured to execute a software program” when atleast one value associated with the software program is stored in aregister that is readable by the processor. In some embodiments,routines executed to implement the disclosed embodiments may beimplemented as part of operating system (OS) software (e.g., MicrosoftWindows®, Linux®) or a specific software application, component,program, object, module or sequence of instructions referred to as“computer programs.”

As such, the computer programs typically comprise one or moreinstructions that are set at various times in various memory devices ofa computer (e.g., computer 40) and which, when read and executed by atleast one processor (e.g., processor 52), cause the computer to performoperations to execute features involving the various aspects of thedisclosure embodiments. In some embodiments, a carrier containing theaforementioned computer program product is provided. The carrier is oneof an electronic signal, an optical signal, a radio signal, or anon-transitory computer-readable storage medium (e.g., the memory 54).

The network interface 46 may include a modem or other interfaces (notshown) for coupling the computer 40 to other computers over the network18. The I/O system 48 may operate to control various I/O devices,including peripheral devices such as a display system 58 (e.g., amonitor or touch-sensitive display) and one or more input devices 60(e.g., a keyboard and/or pointing device). Other I/O devices 62 mayinclude, for example, a disk drive, printer, scanner, or the like.Lastly, the clock system 50 controls a timer for use by the disclosedembodiments.

Operation of a memory device (e.g., memory 54), such as a change instate from a binary one to a binary zero (or vice versa), may comprise avisually perceptible physical transformation. The transformation maycomprise a physical transformation of an article to a different state orthing. For example, a change in state may involve accumulation andstorage of charge or release of stored charge. Likewise, a change ofstate may comprise a physical change or transformation in magneticorientation, or a physical change or transformation in molecularstructure, such as from crystalline to amorphous or vice versa.

Aspects of the disclosed embodiments may be described in terms ofalgorithms and symbolic representations of operations on data bitsstored on memory. These algorithmic descriptions and symbolicrepresentations generally include a sequence of operations leading to adesired result. The operations require physical manipulations ofphysical quantities. Usually, though not necessarily, these quantitiestake the form of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. Customarily,and for convenience, these signals are referred to as bits, values,elements, symbols, characters, terms, numbers, or the like. These andsimilar terms are associated with physical quantities and are merelyconvenient labels applied to these quantities.

While embodiments have been described in the context of fullyfunctioning computers, those skilled in the art will appreciate that thevarious embodiments are capable of being distributed as a programproduct in a variety of forms and that the disclosure applies equally,regardless of the particular type of machine or computer-readable mediaused to actually effect the distribution.

While the disclosure has been described in terms of several embodiments,those skilled in the art will recognize that the disclosure is notlimited to the embodiments described herein and can be practiced withmodifications and alterations within the spirit and scope of theinvention. Those skilled in the art will also recognize improvements tothe embodiments of the present disclosure. All such improvements areconsidered within the scope of the concepts disclosed herein. Thus, thedescription is to be regarded as illustrative instead of limiting.

We claim:
 1. A method of operation of a distributed network systemincluding a plurality of nodes, the method comprising: sending, by afirst subset of the plurality of nodes, a plurality of messages over aprotocol-independent message bus; receiving, by a second subset of theplurality of nodes, the plurality of messages sent over theprotocol-independent message bus; storing content from the plurality ofmessages in a database distributed among at least the second subset ofthe plurality of nodes that received the plurality of messages;publishing the content stored in the database to provide publishedcontent, the published content being accessible by a plurality ofapplications; and accessing the published content by one or more of theplurality of applications to perform one or more functions.
 2. Themethod of claim 1, further comprising, prior to receiving the pluralityof messages by the second subset of the plurality of nodes: propagating,by a third subset of the plurality of nodes, at least some of theplurality of messages sent by the first subset of the plurality of nodesover the protocol-independent message bus in accordance with a floodingexchange.
 3. The method of claim 1, further comprising, prior toreceiving the plurality of messages by the second subset of theplurality of nodes: segmenting the plurality of nodes into a pluralityof clusters, each cluster including one or more nodes of the pluralityof nodes; propagating, among the plurality of clusters, the plurality ofmessages over the protocol-independent message bus in accordance with aflooding exchange.
 4. The method of claim 3, wherein segmenting theplurality of nodes into the plurality of clusters comprises: determiningone or more spanning trees configured to send messages over nodes in aloop-free manner over the protocol-independent message bus, wherein eachof the one or more spanning trees forms a cluster of the plurality ofclusters.
 5. The method of claim 4, further comprising: performing pathtracing to determine a plurality of paths taken by the plurality ofmessages traversing nodes of the plurality of nodes; and preventing asubset of the plurality of messages from traversing a subset of theplurality of clusters when the subset of the plurality of messages haspreviously traversed the subset of the plurality of clusters.
 6. Themethod of claim 1, wherein the plurality of applications comprises atleast one of a routing application, a bandwidth allocation application,or a traffic engineering application.
 7. The method of claim 1, whereinthe plurality of applications are distributed among at least some of theplurality of nodes.
 8. The method of claim 1, wherein the plurality ofapplications are resident on one or more nodes separate and distinctfrom the network system.
 9. The method of claim 1, wherein contentstored in the database is periodically synchronized between nodes of theplurality of nodes that have a common link.
 10. The method of claim 1,wherein the database is a key-value database that does not limit keys orvalues to a maximum size.
 11. The method of claim 1, wherein thedatabase includes existing content associated with one or moretimestamps and the content from the plurality of messages is newcontent, and storing the new content in the database comprises:designating the existing content as expired content upon exceeding aperiod of time stored in the database relative to the one or moretimestamps; and replacing the expired content included in the databasewith the new content from the plurality of messages.
 12. The method ofclaim 1, wherein the database includes existing content and the contentfrom the plurality of messages is new content, and storing the newcontent in the database comprises: designating a subset of the existingcontent to be replaced with the new content; and replacing the subset ofthe existing content with the new content.
 13. The method of claim 1,further comprising: sending a hello message over a common connectinglink from a first node of the plurality of nodes to a second node of theplurality of nodes; receiving an acknowledgement message by the firstnode from the second node, the acknowledgement message acknowledging thereception of the hello message by the second node; sending the hellomessage over a common connecting link from the second node to a thirdnode of the plurality of nodes; and receiving an acknowledgement messageby the first node from the third node, the acknowledgement messageacknowledging the reception of the hello message by the third node. 14.The method of claim 1, wherein the network system comprises a helloprotocol module for sending messages between any two nodes of theplurality of nodes being directly connected to a common link, the helloprotocol module being used to perform any of computing spanning treemetrics, automating clustering of the plurality of nodes, or providingbidirectional discovery of neighboring nodes.
 15. The method of claim 1,wherein the method further comprises: operating the plurality of nodesin a learning mode during which the plurality of nodes learn of publickeys of the plurality of nodes, wherein the plurality of nodes havingpublic keys learned during the learning mode are trusted nodes; andoperating the plurality of nodes in a protected mode during which theplurality of nodes accept messages over the message bus only from thetrusted nodes.
 16. The method of claim 1, wherein the database is akey-value database comprising Thrift-encoded keys and values.
 17. Themethod of claim 1, wherein the message bus utilizes a software frameworkthat provides scalable cross-language services for communicatingmessages among the plurality of nodes.
 18. The method of claim 1,wherein messages sent over the protocol-independent message bus areThrift encoded.
 19. The method of claim 1, wherein the plurality ofmessages comprise a plurality of messages that include content that isencoded in accordance with different coding languages and have differentformats.
 20. The method of claim 1, wherein the network system iscomprised in a backhaul, datacenter, or backbone network.
 21. The methodof claim 1, wherein the one or more functions comprises at least one ofan address assignment function, an auditing function, a monitoringfunction, an adaptive control function, an optimization function, or amachine learning function.
 22. A distributed network applicationframework architecture, comprising: a protocol-agnostic message busconnected to a plurality of nodes of the network, the plurality of nodesbeing configured to facilitate at least one of sending, receiving, orpropagating a plurality of messages over the protocol-agnostic messagebus; a payload-agnostic database distributed across the plurality ofnodes and storing content included in at least some of the plurality ofmessages communicated over the protocol-agnostic message bus; and one ormore applications utilizing the content stored in the database toperform one or more functions.
 23. A node of a distributed networksystem, the node comprising: an interface to a protocol-agnostic messagebus connected to a plurality of nodes of the network system, theinterface being configured to facilitate sending, receiving, andpropagating a plurality of messages over the protocol-agnostic messagebus; a database component of a payload-agnostic database distributedacross the node and the plurality of nodes, the database componentstoring content included in at least some of the plurality of messagescommunicated over the protocol-agnostic message bus; and one or moreapplications utilizing the content stored in the database component toperform one or more functions.